{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 6.6. 卷积神经网络（LeNet）\n",
    ":label:`sec_lenet`\n",
    "\n",
    "\n",
    "通过之前几节，我们学习了构建一个完整卷积神经网络的所需组件。\n",
    "回想一下，之前我们将 softmax 回归模型（ :numref:`sec_softmax_scratch` ）和多层感知机模型（ :numref:`sec_mlp_scratch` ）应用于 Fashion-MNIST 数据集中的服装图片上。\n",
    "为了能够应用 softmax 回归和多层感知机，我们首先将每个大小为 $28\\times28$ 的图像展平为一个 784 固定长度的一维向量，然后用全连接层对其进行处理。\n",
    "而现在，我们已经掌握了卷积层的处理方法，我们可以在图像中保留空间结构。\n",
    "同时，用卷积层代替全连接层的另一个好处是：更简洁的模型所需的参数更少。\n",
    "\n",
    "在本节中，我们将介绍 LeNet，它是最早发布的卷积神经网络之一，因其在计算机视觉任务中的高效性能而受到广泛关注。\n",
    "这个模型是由 AT&T 贝尔实验室的研究员 Yann LeCun 在1989年提出的（并以其命名），目的是识别图像 :cite:`LeCun.Bottou.Bengio.ea.1998` 中的手写数字。\n",
    "当时，Yann LeCun 发表了第一篇通过反向传播成功训练卷积神经网络的研究，这项工作代表了十多年来神经网络研究开发的成果。\n",
    "\n",
    "当时， LeNet 取得了与支持向量机（support vector machines）性能相媲美的成果，成为监督学习的主流方法。\n",
    "LeNet 被广泛用于自动取款机（ATM）机中，帮助识别处理支票的数字。\n",
    "时至今日，一些自动取款机仍在运行 Yann LeCun 和他的同事 Leon Bottou 在上世纪90年代写的代码呢！\n",
    "\n",
    "## 6.6.1. LeNet\n",
    "\n",
    "总体来看，(**LeNet（LeNet-5）由两个部分组成：**)(~~卷积编码器和全连接层密集块~~)\n",
    "* 卷积编码器：由两个卷积层组成;\n",
    "* 全连接层密集块：由三个全连接层组成。\n",
    "\n",
    "该结构在 :numref:`img_lenet` 中所展示。\n",
    "\n",
    "![LeNet中的数据流。输入是手写数字，输出为10种可能结果的概率。](../img/lenet.svg)\n",
    ":label:`img_lenet`\n",
    "\n",
    "每个卷积块中的基本单元是一个卷积层、一个 sigmoid 激活函数和平均池化层。请注意，虽然 ReLU 和最大池化层更有效，但它们在20世纪90年代还没有出现。每个卷积层使用 $5\\times 5$ 卷积核，这些层将输入映射到多个二维特征输出，通常同时增加通道的数量。第一卷积层有 6 个输出通道，而第二个卷积层有 16 个输出通道。每个 $2\\times2$ 池操作（步骤2）通过空间下采样将维数减少 4 倍。卷积的输出形状由批量大小、通道数、高度、宽度决定。\n",
    "\n",
    "为了将卷积块的输出传递给稠密块，我们必须在小批量中展平每个样本。换言之，我们将这个四维输入转换成全连接层所期望的二维输入。这里的二维表示的第一个维度索引小批量中的样本，第二个维度给出每个样本的平面向量表示。LeNet 的稠密块有三个全连接层，分别有 120、84 和 10 个输出。因为我们仍在执行分类，所以输出层的 10 维对应于最后输出结果的数量。\n",
    "\n",
    "通过下面的 LeNet 代码，你会相信用深度学习框架实现此类模型非常简单。我们只需要实例化一个 `Sequential` 块并将需要的层连接在一起。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import paddle\n",
    "from paddle import nn, optimizer\n",
    "\n",
    "class LeNet(nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(LeNet, self).__init__()\n",
    "        self.conv = nn.Sequential(\n",
    "            nn.Conv2D(1, 6, 5, padding=2), # in_channels, out_channels, kernel_size\n",
    "            nn.Sigmoid(),\n",
    "            nn.MaxPool2D(2, 2), # kernel_size, stride\n",
    "            nn.Conv2D(6, 16, 5),\n",
    "            nn.Sigmoid(),\n",
    "            nn.MaxPool2D(2, 2)\n",
    "        )\n",
    "        self.fc = nn.Sequential(\n",
    "            nn.Linear(16*5*5, 120),\n",
    "            nn.Sigmoid(),\n",
    "            nn.Linear(120, 84),\n",
    "            nn.Sigmoid(),\n",
    "            nn.Linear(84, 10)\n",
    "        )\n",
    "\n",
    "    def forward(self, img):\n",
    "        feature = self.conv(img)\n",
    "        output = self.fc(feature.reshape((img.shape[0], -1)))\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "我们对原始模型做了一点小改动，去掉了最后一层的高斯激活。除此之外，这个网络与最初的 LeNet-5 一致。\n",
    "\n",
    "下面，我们将一个大小为 $28 \\times 28$ 的单通道（黑白）图像通过 LeNet。 通过在每一层打印输出的形状，我们可以[**检查模型**]，以确保其操作与我们期望的 :numref:`img_lenet_vert` 一致。\n",
    "\n",
    "![LeNet 的简化版。](../img/lenet-vert.svg)\n",
    ":label:`img_lenet_vert`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LeNet(\n",
      "  (conv): Sequential(\n",
      "    (0): Conv2D(1, 6, kernel_size=[5, 5], padding=2, data_format=NCHW)\n",
      "    (1): Sigmoid()\n",
      "    (2): MaxPool2D(kernel_size=2, stride=2, padding=0)\n",
      "    (3): Conv2D(6, 16, kernel_size=[5, 5], data_format=NCHW)\n",
      "    (4): Sigmoid()\n",
      "    (5): MaxPool2D(kernel_size=2, stride=2, padding=0)\n",
      "  )\n",
      "  (fc): Sequential(\n",
      "    (0): Linear(in_features=400, out_features=120, dtype=float32)\n",
      "    (1): Sigmoid()\n",
      "    (2): Linear(in_features=120, out_features=84, dtype=float32)\n",
      "    (3): Sigmoid()\n",
      "    (4): Linear(in_features=84, out_features=10, dtype=float32)\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "net = LeNet()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "请注意，在整个卷积块中，与上一层相比，每一层特征的高度和宽度都减小了。\n",
    "第一个卷积层使用 2 个像素的填充，来补偿 $5 \\times 5$ 卷积核导致的特征减少。\n",
    "相反，第二个卷积层没有填充，因此高度和宽度都减少了 4 个像素。\n",
    "随着层叠的上升，通道的数量从输入时的 1 个，增加到第一个卷积层之后的 6 个，再到第二个卷积层之后的 16 个。\n",
    "同时，每个池化层的高度和宽度都减半。最后，每个全连接层减少维数，最终输出一个维数与结果分类数相匹配的输出。\n",
    "\n",
    "\n",
    "## 6.6.2. 模型训练\n",
    "\n",
    "现在我们已经实现了 LeNet ，让我们看看[**LeNet在Fashion-MNIST数据集上的表现**]。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.vision.datasets as datasets\n",
    "import paddle.vision.transforms as transforms\n",
    "import sys\n",
    "\n",
    "\n",
    "def load_data_fashion_mnist(batch_size, resize=None):\n",
    "    \"\"\"Download the fashion mnist dataset and then load into memory.\"\"\"\n",
    "    trans = []\n",
    "    if resize:\n",
    "        trans.append(transforms.Resize(size=resize))\n",
    "    trans.append(transforms.ToTensor())\n",
    "    \n",
    "    transform = transforms.Compose(trans)\n",
    "    mnist_train = datasets.FashionMNIST(mode='train', download=True, transform=transform)\n",
    "    mnist_test = datasets.FashionMNIST(mode='test', download=True, transform=transform)\n",
    "    if sys.platform.startswith('win'):\n",
    "        num_workers = 0  # 0表示不用额外的进程来加速读取数据\n",
    "    else:\n",
    "        num_workers = 0\n",
    "    train_iter = paddle.io.DataLoader(mnist_train, batch_size=batch_size, shuffle=True, num_workers=num_workers)\n",
    "    test_iter = paddle.io.DataLoader(mnist_test, batch_size=batch_size, shuffle=False, num_workers=num_workers)\n",
    "\n",
    "    return train_iter, test_iter\n",
    "\n",
    "batch_size = 256\n",
    "train_iter, test_iter = load_data_fashion_mnist(batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "虽然卷积神经网络的参数较少，但与深度的多层感知机相比，它们的计算成本仍然很高，因为每个参数都参与更多的乘法。\n",
    "如果你有机会使用GPU，可以用它加快训练。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "为了进行评估，我们需要[**对**] :numref:`sec_softmax_scratch` 中描述的 (**`evaluate_accuracy`函数进行轻微的修改**)。\n",
    "由于完整的数据集位于内存中，因此在模型使用 GPU 计算数据集之前，我们需要将其复制到显存中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def evaluate_accuracy(data_iter, net):\n",
    "\n",
    "    acc_sum, n = 0.0, 0\n",
    "    with paddle.no_grad():\n",
    "        for X, y in data_iter:\n",
    "            if isinstance(net, nn.Layer):\n",
    "                net.eval() # 评估模式, 这会关闭dropout\n",
    "                acc_sum += (net(X).argmax(axis=1) == y.flatten()).astype('float32').sum().numpy()[0]\n",
    "                net.train() # 改回训练模式\n",
    "            else: # 自定义的模型, 3.13节之后不会用到, 不考虑GPU\n",
    "                if('is_training' in net.__code__.co_varnames): # 如果有is_training这个参数\n",
    "                    # 将is_training设置成False\n",
    "                    acc_sum += (net(X, is_training=False).argmax(dim=1) == y).float().sum().item() \n",
    "                else:\n",
    "                    acc_sum += (net(X).argmax(dim=1) == y).float().sum().item() \n",
    "            n += y.shape[0]\n",
    "    return acc_sum / n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "[**为了使用 GPU，我们还需要一点小改动**]。\n",
    "与 :numref:`sec_softmax_scratch` 中定义的 `train_epoch_ch3` 不同，在进行正向和反向传播之前，我们需要将每一小批量数据移动到我们指定的设备（例如 GPU）上。\n",
    "\n",
    "如下所示，训练函数 `train_ch6` 也类似于 :numref:`sec_softmax_scratch` 中定义的 `train_ch3` 。\n",
    "由于我们将实现多层神经网络，因此我们将主要使用高级 API。\n",
    "以下训练函数假定从高级 API 创建的模型作为输入，并进行相应的优化。\n",
    "我们使用在 :numref:`subsec_xavier` 中介绍的 Xavier 随机初始化模型参数。\n",
    "与全连接层一样，我们使用交叉熵损失函数和小批量随机梯度下降。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def train_ch6(net, train_iter, test_iter, batch_size, optimi, num_epochs):\n",
    "\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "    batch_count = 0\n",
    "    for epoch in range(num_epochs):\n",
    "        train_l_sum, train_acc_sum, n, start = 0.0, 0.0, 0, time.time()\n",
    "        for idx, (X, y) in enumerate(train_iter):\n",
    "            y_hat = net(X)\n",
    "            l = loss(y_hat, y)\n",
    "            optimi.clear_grad()\n",
    "            l.backward()\n",
    "            optimi.step()\n",
    "            train_l_sum += l.numpy()[0]\n",
    "            train_acc_sum += (y_hat.argmax(axis=1) == y.flatten()).astype('float32').sum().numpy()[0]\n",
    "            n += y.shape[0]\n",
    "            batch_count += 1\n",
    "        test_acc = evaluate_accuracy(test_iter, net)\n",
    "        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f, time %.1f sec'\n",
    "              % (epoch + 1, train_l_sum / batch_count, train_acc_sum / n, test_acc, time.time() - start)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "现在，我们[**训练和评估LeNet-5模型**]。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lr, num_epochs = 0.001, 10\n",
    "optimi = optimizer.Adam(parameters=net.parameters(), learning_rate=lr)\n",
    "train_ch6(net, train_iter, test_iter, batch_size, optimi, num_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 小结\n",
    "\n",
    "* 卷积神经网络（CNN）是一类使用卷积层的网络。\n",
    "* 在卷积神经网络中，我们组合使用卷积层、非线性激活函数和池化层。\n",
    "* 为了构造高性能的卷积神经网络，我们通常对卷积层进行排列，逐渐降低其表示的空间分辨率，同时增加通道数。\n",
    "* 在传统的卷积神经网络中，卷积块编码得到的表征在输出之前需由一个或多个全连接层进行处理。\n",
    "* LeNet是最早发布的卷积神经网络之一。\n",
    "\n",
    "## 练习\n",
    "\n",
    "1. 将平均池化层替换为最大池化层，会发生什么？\n",
    "1. 尝试构建一个基于 LeNet 的更复杂的网络，以提高其准确性。\n",
    "    1. 调整卷积窗口大小。\n",
    "    1. 调整输出通道的数量。\n",
    "    1. 调整激活函数（如 ReLU）。\n",
    "    1. 调整卷积层的数量。\n",
    "    1. 调整全连接层的数量。\n",
    "    1. 调整学习率和其他训练细节（例如，初始化和周期数）。\n",
    "1. 在 MNIST 数据集上尝试以上改进的网络。\n",
    "1. 显示不同输入（例如毛衣和外套）时，LeNet 第一层和第二层的激活值。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 2.1.2 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
